Improving integration quality for heterogeneous data sources

This work considers a problem of integrating heterogeneous semi–structured data sources with the purpose of estimating integration quality (IQ). During the integration of such data sources the IQ estimation plays an important role, because correspondences and dependencies within and across the sources are not completely known, the schema or semantics might be missing, which leads to results with unpredictable trustworthiness. Therefore, we consider existing methods of analysis of such data sources and investigate a possible scenario of the integration process. We analyze a problem of uncertainty in the integration process. For that we introduce examples demonstrating present inability of accounting for the combined uncertainties affecting integration quality. We introduce a classification of the types of uncertainties. In order to account for the uncertainties we suggest using the statistical method Latent Class Analysis (LCA), related to the Latent Variable Models. This method allows to analyze the influence of the latent factors on the set of data. As related to the task of integration, by a latent factor we understand belonging of an object to a real–world class and in its turn the role of LCA is to interpret correlation of discovering identical objects from different data sources as a display of that universal factor. We build a statistical model of the integration task, i.e., draw correspondences between the terms of statistics and the terms of integration. Presence of at least three data sources is necessary for making use of LCA, at that, when integrating two sources, an integrated database itself can represent a lacking third source. The result of the analysis is the probability value of the real–world class membership for a considered group of objects. Derived by LCA real–world class membership value includes influence of all types of uncertainties and reflects IQ. By applying LCA to each triplet of the corresponding classes at the lowest schema level and obtaining real–world class membership, we can calculate the support of the real–world class for any level of the database, including database itself, as a weighted average of the real–world class membership for all classes at the lowest level. The proposed approach does not solve common problems of integration of the heterogeneous data sources, but rather can be used for evaluating and improving IQ. Capability to evaluate the IQ gives an important tool to the users concerned with the data’s trustworthiness. It helps them to answer the